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■In 2009-10, the Urban Institute and The Center 
for What Works developed and tested an outcome 
measurement framework with DC OST providers. 

■Providers offered a wide range of activities, 
operated in different settings, and served 
4 youth age 5 to 24. 



■ The challenge ahead is to expand the pilot 
to all OST programs in the District. 




D.C. Out-of-School Time Programs 
Tackle Outcome Measures 

Mary K. Winkler, Carol J. De Vita, Saunji Fyffe, and Debra B. Natenshon 



F unders and nonprofit leaders are increas- 
ingly looking for information that 
demonstrates the achievements of service 
programs. The questions “What services do 
you provide?” and “How many people do you 
serve?” are being replaced by “What have your 
services accomplished?” This results-oriented 
environment set the stage for a pilot study in the 
District of Columbia to measure outcomes of 
nonprofit out-of-school time (OST) programs. 

Background 

In summer 2009, the D.C. Children and Youth 
Investment Trust Corporation (The Trust) was 
engaged in an ongoing effort to build the internal 
capacity of its OST grantees and help strengthen 
and improve the quality of services for the 
Districts children and youth. At the same time, 
the Urban Institute (UI) and The Center for 
What Works (CWW) were looking for partners 
to develop and test an outcome measurement 
framework with local OST providers. This over- 
lap of interests and goals led to a pilot project 
(January-June 2010) to train a sample of the 
Trusts OST grantees on collecting and reporting 
outcome data. 



Recruitment of grantees for the pilot study 
began in fall 2009. A questionnaire was distrib- 
uted to all 74 Trust OST grantees for fiscal year 
2010 to measure their interest in and readiness 
for participating in the pilot study. UI and 
CWW also made presentations at the Trust’s fall 
grantee meeting to explain the purpose of the 
pilot and answer questions about the pilot’s 
procedures and requirements. A formal selection 
process that included the questionnaire results 
and feedback from the grantees and the Trust 
was established, and 27 grantees were selected for 
the pilot. 

Who Participated 

The organizations selected for the pilot repre- 
sented a diverse cross-section of the Trust’s OST 
grantees: 

• They offered a wide range of after-school 
activities. Some provided academic tutoring 
and mentoring programs; others gave computer 
skills instruction; still others offered dance, arts, 
culinary, and recreational programs. 

• They operated in different settings including 
public charter schools, community-based non- 
profit groups, and faith-based organizations. 



• They focused on children and youth of 
various ages between 5 and 24. 

• They operated programs of different sizes 

serving as few as 25 and as many as 100 or 
more youth. Overall, more than 800 youth 
were tracked during the pilot. 



Participants in the 
pilot study were 
eager to learn how to 
better manage their 
programs, keep young 
people engaged, and 
demonstrate their 
accomplishments . 
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• They had mixed prior experience with 
program evaluations or performance measure- 
ment. About 23 percent reported considerable 
experience, 50 percent reported moderate 
experience, and 27 percent reported very little 
experience. 

Activities and Structure of the Pilot Study 

UI and CWW conducted training events and 

offered technical assistance to give pilot partici- 
pants the knowledge, direction, and tools they 

would need to collect and report outcome data. 

Activities included the following: 

• A half-day workshop in January introduced 
participants to the concepts of outcome 
measurement, helped them identify suitable 
outcomes for their programs, and gave partici- 
pants tools (such as survey instruments and 
Excel worksheets) as well as technical assistance 
for collecting and entering data. The UI/CWW 
team prepared a guidebook with instructional 
materials and gave each participant a copy to 
use as a reference and share with other members 
of his or her staff. 

• Two webinars guided pilot participants 
throughout the process. The first webinar in 
February reviewed data collection and data- 
entry procedures; the second webinar in 
April/May focused on data analysis and 
reporting. The webinars provided opportuni- 
ties for grantees to ask questions and learn 
from one another. 

• Individual technical assistance was provided 
by the UI/CWW team throughout the pilot. 
Most queries were handled by e-mail and 
telephone. 

• Two rounds of data collection were required 
of participating grantees. Initial baseline data 
were collected in February, and follow-up data 
were collected in May. Although the length of 
time between the first and second rounds of 
data collection was too short to measure reli- 
able program outcomes, the grantees learned 
how to conduct pre- and post-program assess- 
ments that will help them better understand 
the progress youth are making. 

• A post-pilot survey provided UI/CWW and 
the Trust with important feedback on what 
participants learned and what they found 



helpful about the pilot. This information will 
be instrumental in strengthening future out- 
come measurement efforts for community- 
based OST providers. 

Key Findings 

At least five key findings emerged from this six- 

month pilot program. 

1. Participants in the pilot were eager to learn. 

They typically were drawn to the pilot 
because they wanted to learn how to better 
manage their programs, keep young people 
engaged, and demonstrate their accomplish- 
ments to funders. At the initial workshop, 
several grantees eagerly expressed interest in 
collecting all 12 of the suggested indicators. 
The UI/CWW team counseled these enthu- 
siastic participants to start slowly, learn the 
skills and data collection procedures, and 
expand into more areas later. Grantees typi- 
cally selected between three and eight indica- 
tors to track. They tended to favor indicators 
measuring academic performance and school 
engagement. 

2. Providing ready-made tools facilitated 
grantees’ learning curve. Community-based 
service providers often have little time to 
locate or develop assessment tools. To address 
this problem, the UI/CWW team provided 
not only a menu of potential outcome indica- 
tors that grantees were invited to select and 
track, but also a detailed Excel spreadsheet to 
record the data collected during the pilot. 
Most grantees found the spreadsheet effective 
and easy to use. In fact, one of the more 
experienced grantees said that receiving the 
spreadsheet at the initial workshop saved 
her hours of time in setting up her own 
reporting template. A few of the more 
advanced grantees, however, requested other 
software options. 

3. Technical ability varied greatly among 
participants. Grantees with more advanced 
technical skills had little problem collecting 
and recording data. Those with limited or no 
experience with Excel had a more difficult 
time. Targeting training materials to grantees 
with a wide range of skill levels was a chal- 
lenge. It was difficult to determine grantee skill 



levels before the training because organizations 
tended to over- or underestimate their actual 
skills and experience in collecting and report- 
ing data. 

4. Providing technical assistance was critical to 
the success of the pilot. Participants required 
assistance beyond the formal workshop and 
webinars. Pilot participants needed help, for 
example, with basic Excel functions, complet- 
ing the reporting template, and collecting 
and coding data. Most of the one-on-one 
technical assistance was conducted by phone 
or e-mail correspondence. Personalized assis- 
tance kept grantees focused and on track. 

5. Prior experience with outcome measure- 
ment led to different needs and expecta- 
tions about training materials. For example, 
grantees with little to no experience measur- 
ing performance-related outcomes rated the 
webinars high. Grantees with more experi- 
ence tended to rate the webinars lower. This 
variation in experience suggests that commu- 
nity-based organizations with prior experience 
in outcome measurement may need more 
advanced training. Several grantees suggested 
greater focus on the internal and external uses 
of the data and help with communicating 
their successes to staff, board members, and 
outside funders. Organizations with more 
limited exposure to outcome measurement 
require basic training on how to collect, 
record, analyze, and report information. 

Areas for Improvement and 
Recommendations 

Although the pilot was a limited, six-month 
effort to refine and test a coherent strategy to 
introduce outcome measures to community- 
based nonprofits and build their capacity to 
collect and report such data, it is clear that 
several steps need to be taken to enhance and 
support future efforts. Our initial thoughts are 
outlined below. 

• It takes at least a full academic year of data 
collection to obtain reliable measures of 
change. The post-pilot survey revealed that 
the timing and length of the pilot was a 
major concern for participating grantees. The 
pilot began in the middle of the school year 
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(January) and ended before die last day of 
school (May). This relatively short time frame 
challenged grantees to fit new data collection, 
data entry, and reporting procedures into 
already-full schedules, and some measures, 
such as final grades, were not available by the 
end of the pilot. Several grantees suggested 
extending the pilot beyond six months to 
cover the full academic year. 

• Getting school records was a stumbling block 
for many grantees. Although many grantees 
wanted to measure the academic performance 
of the young people they were working with, 
they encountered difficulty in obtaining these 
records. Student report cards and standardized 
test scores are not readily released to commu- 
nity-based OST providers, making it difficult 
for them to measure and assess the needs and 
academic progress of the young people in their 
program. Some pilot participants tried to 
obtain academic records through parents, but 
this strategy yielded mixed results. Supporting 
the current plan to share student data through 
after-school coordinators in public schools is a 
vital step in helping measure the outcomes of 
youth and youth-serving programs, but appro- 
priate safeguards must be in place to protect 
student privacy. 

• Grantees expressed a need for age-appropri- 
ate assessment instruments. Grantees serving 
young children found the assessment tools 
too advanced cognitively for some of their 
participants. Based on this feedback, the 
UI/CWW team recommends that further 
work be done to identify valid assessment 
tools that target children in elementary and 
middle schools. 

• In a few cases, literacy deficiencies made 
data collection difficult. Some grantees 
reported that youth found the wording of the 
assessment questions confusing and difficult 
to understand. Some youth were not able to 
distinguish between categories of survey 
responses and therefore were unsure how to 
answer questions. The adults administering 
the assessments had to help these young 
people complete the questionnaires. Such 
assistance can skew results and is not desired 
to collect valid information. The situation 



illustrates one difficulty in conducting perform- 
ance assessments among youth with varying lev- 
els of education and reading comprehension. 
Materials used in the pilot were drawn from 
previously validated assessment tools, but they 
proved inadequate for some youth. Further 
research is needed to develop tools that might be 
used for young people with low levels of literacy. 

A Successful Effort 

This pilot program conducted by UI and CWW 
for the Trust was designed to build the capacity 
of a small group of Trust OST grantees and 
enable them to collect and report program out- 
comes. By several measures, the pilot was highly 
successful. 

• 25 of 2 7 organizations completed the pilot 
program, for a 93 percent retention rate. One 
organization withdrew because it had an 
opportunity to participate in a pilot focused on 
fundraising and it did not have the capacity to 
do both pilots; the other organization with- 
drew because of unanticipated organizational 
challenges. 

• Participants were highly engaged in all core 
pilot activities; the overall participation rate 
was 84 percent. 

• Over 95 percent of grantees said they would 
recommend the pilot to colleagues or other 
organizations. 

• The majority of participating organizations (73 
percent) planned to continue using the pilot’s 
indicators and data collection techniques. The 
remaining 27 percent said they were unsure of 
future plans. No organizations had a negative 
response. 

The results of this pilot reflect a great 
beginning. Community-based groups serving 
the District’s children and youth are clearly 
receptive and eager to learn. The challenge 
ahead will be to expand this work to all OST 
programs in the District and beyond. What is 
needed is a carefully structured training and 
technical assistance program to ensure that these 
skills become infused into the culture of non- 
profit organizations so programs can measure 
their results, adjust their activities if needed, and 
achieve favorable outcomes for youth. • 



In Their Own Words 



Here’s what DC OST providers said about 
the outcome measurement pilot. 

This pilot helped us organize our data effi- 
ciently and helped us think through appropri- 
ate “indicators” to focus on. Also, having this 
reporting template is tremendously useffilfor 
us as we continue to refine and strengthen our 
evaluation tools. 

We have always struggled with finding 
quantitative indicators to assess self- 
esteem! self-efficacy. Since one of our program 
aims is to improve self-esteem for girls, this 
was quite a challenge. Through participation 
in this pilot, we were able to obtain indica- 
tors to assess our self-esteem outcomes. 

The pilot was excellent in determining out- 
comes, and I like the idea of showing us how 
we could use this data in proposal-writing. 
Thanks! 

I like that we are working collectively on 
developing measures that can be used by many 
youth development organizations to help us 
build best practices and standards for meas- 
urement and evaluation. 

Very well organized pilot. I also learned some 
things that I can use in the future assessments 
with our students. Thank you. 

Thanks for including us in the pilot! This 
process was extremely helpful for us in assess- 
ing some of our outcomes. The databases were 
also great! We can use some of the formulas for 
other indicators that we use in our programs. 

I feel your work will have a strong impact for 
us in the future. It made us look back and 
reflect on improving the direction we need to 
go to make our case and find options to 
include those who for whatever reason don’t 
want to be involved. 
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About the Urban Institute and The Center for What Works 

The Urban Institute is a nonprofit, nonpartisan policy research and educational organization 
located in Washington, D.C., that examines the social, economic, and governance challenges 
facing the nation. It provides information, analyses, and perspectives to public and private deci- 
sion makers to help them address these problems and strives to deepen citizens’ understanding of 
the issues and trade-offs that policymakers face. Since the 1970s, the Urban Institute has been a 
pioneer in developing outcome and performance measures for the nonprofit and public sectors. 

The Center for What Works is a nonprofit organization dedicated to building performance 
measurement capacity and developing a common language for the social sector to benchmark 
results, learn and improve. Founded over 15 years ago with a vision of helping nonprofits bench- 
mark their results, WhatWorks serves as a bridge between rigorous research and practical tools, 
resources and services for nonprofits and foundations to become more effective. 
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